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Inside This Report 


Colleges and universities across the United States are working to enhance academic 
advising to improve students’ experiences and outcomes. These improvements 

require evidence-based redesigns of policies, practices, and technical and nontechnical 
advising tools. As institutions undertake redesigns, much of the focus is on leveraging 
technology to support advisors in communicating with students, identifying students 
who are struggling and supporting them with timely and appropriate interventions, and 
facilitating meaningful and productive advising sessions. 


This report presents findings on the implementation ofa technology-mediated advising 
redesign within the context of arandomized controlled trial (RCT) at three institutions— 
the University of North Carolina at Charlotte (UNC Charlotte), California State University 
at Fresno (Fresno State), and Montgomery County Community College (MCCC). These 
institutions were participants in a broader initiative called iPASS (Integrated Planning 

and Advising for Student Success), which provided support to 26 institutions adopting 
technology-based advising redesigns. UNC Charlotte, Fresno State, and MCCC aimed to 
improve their advising systems using a three-pronged approach: expanding informational 
communications to students, identifying and supporting students who are struggling, and 
improving advising sessions. 


Though the RCT showed that quantitative student outcomes, such as grade point average 
(GPA) and credits earned, did not significantly change during the first two semesters of the 
three-semester study (Mayer et al., 2019), qualitative fieldwork suggests that stakeholders 
across the three institutions gained valuable experiences that may provide lessons for 
other colleges considering engaging in advising redesign. In this report, we explore how 
advisors adjusted their approach to advising work, how they used relevant tools and 
resources, and some of the challenges that arose during the redesign’s implementation. 
The following are highlights from our qualitative research: 


e The colleges were able to implement the intended improvements to advising. Most 
advisors leveraged new and existing technologies to improve communications to 
students and implement new advising practices during face-to-face sessions. 


e The colleges found it challenging to engage students with informational messages, 
such as reminders about upcoming deadlines and information about campus 
resources, especially when using mass communications that were not personalized. 


e Early-alert data from faculty and self-reported data from students were hard to gather 
but useful, to a degree, for guiding interventions with individual students. 


e Advising technologies offered some efficiencies in scheduling and documenting 
advising sessions. These technologies also enhanced degree and career planning 
discussions. 


e Through the use of supporting materials, advisors learned new ways or new language 
to use to engage students during advising sessions. Advisors appreciated these 
resources, provided that they were given the discretion to determine whether and 
when to apply the suggested strategies. 
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Introduction 


There is a broad consensus that advising can help college students make good decisions as they select 
their majors and courses and help them stay on track to achieve their goals (Center for Community 
College Student Engagement [CCCSE], 2018; Karp, 2011; Montelongo, 2003). Engaging with 

an advisor —particularly early on in students’ college experience—positively affects students’ 
engagement, decision-making, and self-confidence (CCCSE, 2018). Though some studies have 
found the association between advising and student outcomes to be nonsignificant, recent studies 
suggest that the use of advising services is positively associated with credit completion and grade 
point average (GPA) (Kot, 2014; Young-Jones, Burt, Dixon, & Hawthorne, 2013; Pascarella & 
Terenzini, 2005). These findings point to the important role advisors play in helping students have a 
positive experience on campus and stay on track toward the attainment of their postsecondary goals. 


To maximize the benefits of advising, institutions across the country are redesigning their advising 
processes and implementing new advising technologies. Community colleges and broad-access 
four-year colleges, in particular, tend to lack the resources to hire more advisors, so they often look 
to use technology to better understand students’ needs and help advisors meet them. Advising 
technology has the potential to make advisors’ communications with students more efficient, 
useful, and frequent. Moreover, some advising technologies have data capabilities that can enable 
advisors to intervene strategically with students who need support. For example, early-alert 
systems allow faculty to identify students who are missing assignments or showing other signs of 
academic struggle. When early alerts are issued, advisors can reach out to struggling students to offer 
assistance. Additionally, advisors can use early-alert data during advising sessions to guide their 
conversations with students. 


Since 2015, 26 institutions nationwide have redesigned their advising structures and processes 
using technology as part of the Integrated Planning and Advising for Student Success (iPASS) 
initiative, led by EDUCAUSE with support from the Bill & Melinda Gates Foundation and 

the Helmsley Charitable Trust. iPASS grantees have implemented or enhanced a range of 

advising technologies, including early-alert, degree planning, and communication tools and 
case-management software. Many have also adopted new practices in order to offer higher quality 
support to students. For more information on iPASS and the Community College Research Center’s 
(CCRC) role in the initiative, see the accompanying text box. 


CCRC and MDRC partnered to study the implementation and impact ofan enhanced iPASS 
initiative at three of these colleges—the University of North Carolina at Charlotte (UNC Charlotte), 
California State University at Fresno (Fresno State), and Montgomery County Community College 
(MCCC). These three institutions were selected because they were well poised to align their advising 
reforms with the parameters of the study and interested in measuring the impact of the iPASS 
enhancements. The study employed a randomized controlled trial (RCT) as well as site visits and 
interviews. Ina previous report (Mayer etal., 2019), the research team detailed early findings from 
the RCT, discussed whether institutions implemented the enhanced iPASS model with fidelity 

to its design, and described how the experience of students in the program group contrasted with 
that of students in the control group. The current report focuses on insights gained from studying 
the experiences of the three institutions in implementing the enhanced iPASS model, which may 
be useful to other institutions engaging in advising redesign. We also share advisor and student 
perspectives on the advising practices and technologies that were part of the redesign. 
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CCRC’s Role in Three iPASS Research Projects 


The Integrated Planning and Advising for Student Success (iPASS) initiative—which has provided up to 
$225,000 to each of 26 colleges to help them adopt technologies for improving education planning, advising, 
and student risk targeting and intervention by 2018—was launched in 2015 with funding from the Bill & 
Melinda Gates Foundation and The Helmsley Charitable Trust. It followed on the heels of a similar initiative, 
undertaken from 2012 to 2015 at 19 colleges, in which several lessons were learned: 


e Emerging technologies have the potential to allow students to create and follow academic plans 
effectively, receiving support when they struggle. 
e Technology alone is not enough to achieve project goals. Deep changes in institutional structures, 
systems, and attitudes are required. 
e High-quality advising and student support may be facilitated through a set of core SSIPP principles, 
which call for advising to be sustained, strategic, integrated, proactive, and personalized. 
CCRC has been involved in both initiatives. Under the more recent initiative, EDUCAUSE and Achieving the 
Dream (ATD) have provided implementation services in the form of technical assistance to iPASS grantee 
colleges, while CCRC has conducted research on college activities and the student experience. All three 
organizations—EDUCAUSE, ATD, and CCRC—have sought to learn whether the reform of advising and 
student supports—made possible through the use of technology—provides students with a more seamless 
and holistic advising experience and ultimately improves student outcomes. 


Asan evaluator and thought partner in the 2015-2018 iPASS initiative, CCRC has been engaged in three 
related research projects, which have resulted in reports, presentations, blogs, tools, and other resources for 
the field. 


Project 1. Measuring trends in development and scaling: CCRC has analyzed progress in 
implementation and student outcomes during the grant period across all 26 participating colleges. Resulting 
reports include a survey of technology use and advising practices provided to the colleges, a baseline report 
of key performance indicators (KPIs) (Armijo & Velasco, 2018), and a final report of trends in the KPls after 
two years of project implementation (Velasco, Hughes, & Barnett, 2020). 


Project 2. Understanding implementation: CCRC has studied implementation processes at nine colleges, 
some of which emphasized advising in STEM pathways. We conducted a review of the literature (Fletcher, 
Grant, Ramos, & Karp, 2016), reported on the use of predictive analytics (Klempin, Grant, & Ramos, 
2018), released a set of case studies of four iPASS colleges (Klempin, Pellegrino, Lopez, Barnett, & 
Lawton, 2019), and studied how iPASS reform has unfolded at different levels of the college ecosystem 
(Klempin & Pellegrino, 2020). We also wrote an invited chapter on the SSIPP principles in practice 
(Klempin, Kalamkarian, Pellegrino, & Barnett, 2019). 


Project 3. Evaluating enhanced advising at three colleges: |n collaboration with MDRC, CCRC has 
conducted research at three colleges that were provided technical assistance as they developed enhanced 
iPASS advising systems targeted to specific student populations. We partnered in an evaluation that included 
arandomized controlled trial and qualitative fieldwork to understand implementation at each college. This 
resulted in a report on the project designs developed at each college (Kalamkarian, Boynton, & Lopez, 
2018), an interim report on early outcomes (Mayer et al., 2019), a report on implementation (current report), 
and a final report on outcomes (Miller et al., in press). 


a ~ COMMUNITY COLLEGE RESEARCH CENTER | TEACHERS COLLEGE, COLUMBIA UNIVERSITY 


Background 


The Rationale for Redesigning Advising 


Atcommunity colleges and broad-access four-year colleges and universities, where 
graduation rates remain low, there is a growing emphasis on strengthening advising 

and student services to support students in persisting toward a credential (Himes & 
Schulenberg, 2016). There is some evidence that advising improves student outcomes and 
experiences. For example, Kot (2014) used a propensity score matching analysis to compare 
the outcomes of first-year, first-time-in-college students who met with an advisor with 
those of similar students who did not meet with an advisor. Students who met with an 
advisor achieved higher first-term, second-term, and cumulative first-year GPAs. CCCSE 
(2018) found that community college students who met with an advisor were more engaged 
in campus activities, faculty interactions, and learning activities and supports than their 
peers who did not meet with an advisor. Further, community college students who met with 
an advisor made decisions early in their tenure at college—such as enrolling in courses that 
aligned with their career goals—that strengthened their self-confidence. 


However, advisors at such institutions experience substantial time and resource constraints. 
Some have as many as 1,200 students in their caseload, making it challenging for them to 
engage with students beyond helping them register for courses for the following semester 
(CCCSE, 2018). With such high caseloads, advisors have limited time to explore students’ 
academic and career goals, conduct multisemester course planning, or discuss nonacademic 
circumstances that may be affecting students’ progress (Kalamkarian & Karp, 2015). High 
caseloads also make it more difficult for advisors to be proactive in reaching out to students, 
making it more likely that the students who receive advising are not those with the greatest 
need but those with the know-how and confidence to seek out support. Faced with these 
issues, postsecondary institutions are increasingly redesigning advising to make it more 
accessible and proactive, with the ultimate goal of improving student outcomes. 


Defining High-Quality Advising 


At colleges participating in the iPASS initiative, advising redesign is based on two 
complementary ways of framing high-quality advising practices and systems. From an 
advising-as-teaching perspective (Appleby, 2008), effective advisors teach students 
how to navigate college and encourage the development of noncognitive skills such as 
resourcefulness, professionalism, and accountability. In addition to helping students 
register for courses, advisors help students clarify their educational goals and understand 
the longer term implications of their academic decisions. Advisors also teach students 
how to address any academic hurdles they encounter (Appleby, 2008; Habley, Bloom, & 
Robbins, 2012; National Academic Advising Association, 2006). 


High-quality advising can also be conceived of in terms of the SSIPP principles (Karp, 
Kalamkarian, Klempin, & Fletcher, 2016). In this framing, effective student support 
is not structured as a one-time intervention but sustained throughout students’ time 
at college. Advisors are strategic, differentiating their efforts based on students’ needs. 
Advising and student supports are integrated into the student experience, and advisors 
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and student support staff are proactive in reaching out to students. Finally, advisor— 
advisee engagement is personalized, so students get help from advisors who know them as 
individuals (Karp & Stacey, 2013; Kalamkarian, Boynton, & Lopez, 2018). 


The iPASS initiative is focused on using technology to help academic institutions provide 
high-quality advising through increased data and efficiencies. For example, advising 
technologies may make it easier to identify students who are struggling and deliver them 
targeted interventions, providing overburdened advisors with the information and 
resources they need to be efficient and strategic with their services. 


Advising Enhancements at Fresno State, UNC Charlotte, and 
MCCC 


The three institutions in our RCT study implemented an enhanced version of the iPASS 
initiative over three semesters for a subset of randomly assigned students (the program 
group). While all three had launched advising technologies as part of their broader 
iPASS work, for the RCT study, they furthered their advising redesigns by developing 
and implementing systematic processes in three main areas: expanding informational 
communication to students, identifying and supporting struggling students, and 
improving advising sessions. 


e Toexpand informational communication, colleges increased email and phone 
outreach to inform students about campus resources and activities, such as ways to 
access tutoring and deadlines for financial aid requirements. 


e To proactively identify and support struggling students, each college deployed a 
student self-analysis survey at the beginning of the semester. Each also utilized 
early-alert software programs that enabled faculty to notify students and their 
advisors when students were experiencing academic challenges in specific courses; 
advisors and support staff could then conduct targeted outreach to the students who 
were flagged. Finally, each college required at least some students in the program 
group to meet with an advisor to receive additional support and clarify their academic 
and career pathways. 


e To facilitate improved advising sessions, each college, with support from the research 
team, developed an advising toolbox, or a document that advisors could use as a guide 
when meeting with students face-to-face. The toolbox contained learning objectives 
for the advising session; probing, open-ended questions to foster student-led 
discussion; and suggested topics for advisors to explore with students, including 
noncollege obligations, career aspirations, and experiential learning opportunities 
such as study abroad. 


To implement these reforms, UNC Charlotte, Fresno State, and MCCC engaged both 
existing and new staff. Each institution identified at least one advisor or advising director to 
assume leadership of the advising redesign project. At Fresno State, for example, one advisor 
and one interim advising director were referred to as “iPASS coordinators,” overseeing all 
of the administrative functions associated with the iPASS grant. At all three institutions, 
project leads were critical to the implementation of redesigned strategies and practices, 
issuing ongoing guidance and reminders to advisors. 
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Implementing redesigned advising practices also required the efforts of academic advisors at 
each campus. UNC Charlotte and Fresno State have advising centers at each of their colleges 
(e.g., colleges of education and business); pre-major and major students receive advising at their 
college, and undeclared students do so at a centralized advising center. At MCCC, academic 
advising is offered by the Student Success Center; advisors are assigned programs of study and 
advise students based on their intended or declared program. Because the redesign targeted 
students pursuing a wide range of programs of study, most advisors at each of the three 
institutions participated in implementing redesigned practices. See Appendix Table A2 for 
more information on the target populations at each college. 


Both Fresno State and MCCC created new positions and hired new staff to fill those 

positions to implement their redesigns. Fresno State hired two peer mentors to disseminate 
informational emails and call students flagged on early-alert surveys for targeted guidance. 
MCCC created the position ofa retention specialist. Previously, the college did not have 
services to support students with acute nonacademic challenges, and the retention specialist 
was hired to meet that need. Advisors were able to refer students with nonacademic issues that 
they felt required more intensive support to the retention specialist. 


Qualitative Study Design 


Our implementation research focused on three questions: 


1. How did colleges implement the three main elements of the advising redesign? 
a. How did colleges expand informational communications? 
b. How did colleges identify and support struggling students? 
c. How did colleges improve advising sessions? 
2. How did colleges create broader policies and structures to support the implementation 
of the advising redesign? 
3. What are the implications for other colleges considering advising redesign? 


To address these questions, we conducted two site visits at each participating institution from 
spring 2017 to fall 2017. During each visit, we interviewed a range of stakeholders, including 
administrators, advisors, key project staff (staff who managed the design and implementation 
of the project), other relevant student support staff (such as peer mentors at Fresno State), and 
students. All interviews were audio-recorded and transcribed. The research team also took 
detailed notes and completed a debrief after each site visit to capture their initial impressions. 


To understand the institutions’ experiences during the final semester of implementation 
(spring 2018), we conducted phone interviews with project leads and a subset of 
advisors from each institution. Project leads and advisors were again asked about their 
implementation experiences and perceptions of the redesign. In addition, we asked what 
aspects of the redesign they would like to see further scaled and why, and what they were 
planning next for the redesign. 


Our data are derived from 238 in-person interviews! and 18 phone interviews, which we 
analyzed using Dedoose qualitative analysis software. The research team used an a priori 
coding scheme whereby we began with an initial set of codes based on the components of the 
redesign, such as informational outreach and advising sessions.” 
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Our data are limited in a few ways that are important to note. First, our data collection did not 
include faculty. Consequently, the faculty perspective on redesign strategies—particularly on 
ways to identify and support students struggling in their courses—is missing in our data. Second, 
advising redesign takes time, and stakeholders’ experiences and perspectives likely continued to 
evolve beyond the scope of our data collection period. Third, our data are limited to individuals 
who were willing to participate in interviews, and interviewees may not be representative of 

the broader population of stakeholders affected by the redesign. Finally, contextual factors 

can influence reform implementation in various ways, so other colleges should take their own 
contexts into consideration when applying lessons from the current report. Characteristics of the 
three colleges participating in the current study are shown in Table 1. 


Table 1. 
Institutional Characteristics 
CHARACTERISTICS FRESNO STATE i terete] UNC CHARLOTTE 
Urbanicity Large city Large suburb Large city 
Degrees awarded Bachelor’s and graduate Associate Bachelor’s and graduate 
Open admission No Yes No 
Total students 24,405 11,480 28,721 
Total undergraduates 21,530 11,480 23,404 
Undergraduate enrollment status (%) 
Full-time 87.5 32.9 86.6 
Part-time 12.5 671 13.4 
Race/ethnicity of undergraduates (%) 
American Indian or Alaska Native 03 0.3 0.3 
Asian 14.3 5.7 6.0 
Black or African American 3.0 14.3 164 
Hispanic 49.3 6.2 9.2 
Native Hawaiian or other Pacific Islander 0.2 0.3 O41 
White 19.9 59.3 58.8 
Two or more races 2.8 3.0 4.2 
Race/ethnicity unknown 47 9.3 26 
Nonresident alien 5.6 16 24 
Financial aid status of undergraduates (%) 
Awarded Pell Grant 574 27.5 37.3 
Full-time, first-time students awarded any aid 874 63.7 75.9 


Retention rate* (%) 


Full-time 79.0 65.0 82.0 

Part-time 470 48.0 79.0 
Completion rate of degree/certificate® (%) 

100% of normal time 15.0 8.0 25.0 

150% of normal time 52.0 20.0 55.0 

200% of normal time 60.0 26.0 58.0 


Note. Data from U.S. Department of Education, National Center for Education Statistics, IPEDS 2016-17. 
First-to-second-year retention rates of students who enrolled in college for the first time in fall 2016. 


’Completion rates are calculated for two-year degrees or certificates for MCCC and four-year bachelor’s degrees for UNC Charlotte 
and Fresno State. Normal time for two-year degrees or certificates is defined as two years. Normal time for four-year bachelor’s 
degrees is defined as four years. 
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Lessons From the Three Colleges 


While there were similarities in the strategies the institutions used to expand 
informational communication, identify and support struggling students, and improve 
advising sessions, each institution used a unique approach. In the sections that follow, we 
describe the enhancements the colleges made in these three areas and outline the main 
lessons other colleges can take away from our findings. 


Expanding Informational Communication 


All three colleges aimed to enhance their communications to students about general topics, 
such as campus services or tips for exams, and increased their informational phone calls 

or emails to program group students throughout the semester (as detailed in Table 2). The 
research team collaborated with advising redesign leaders at each institution to develop 
messages before the first semester of the project. Over time, school leaders and advisors 
modified their messages to refine the tone, personalize the wording, and include resources 
they believed were relevant to particular groups of students. In our interviews, several lessons 
emerged related to the informational communications portion of the advising redesign. 


Table 2. 
Summary of Strategies to Expand Informational Communication 
FRESNO STATE MCcCC UNC CHARLOTTE 
e Atthe start of the semester, peer e Advisors and project leads sent e Advisors notified students in the 
mentors emailed and called students messages informing students in program group of their enrollment in 
in the program group about upcoming the program group of deadlines, critical progression courses via email 
degree planning workshops and resources, and other topics. They in Week 2 of the semester. Notification 
appointments. sent messages approximately every emails included reminders about 
two weeks, sending out a total of eight support services. 
messages. 


Note. A critical progression course is a course considered predictive of success in a major, such as Principles of Accounting for a 
business major or General Chemistry for a biology major. 


Advisors valued the more consistent and frequent communications with 
students, but these communications did not often engage students. 


At MCCC and UNC Charlotte, program group students received messages from an 
academic advisor containing general information about the college every two weeks 
during the first semester of implementation. Advisors at both institutions noted that these 
communications were an important addition to their more personal outreach. An advisor 
from MCCC explained that these messages “are really good at highlighting the kind of 
information that we end up helping students with most of the time.” From this advisor’s 
perspective, the college’s informational messages were especially useful for reaching 
students who might have difficulty navigating online resources independently. UNC 
Charlotte’s communications included informational tables designed to guide students to 
college resources (See Figure 1 for an example.) 
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Figure 1. 
Table Utilized in Informational Communications at UNC Charlotte 


UNC Charlotte Advisor & 
to Student 


IF YOU WANT... b Col UE), [o] 0) > im 


Help making an action plan that works for 


you, or if you’re not sure what to do Maer withime 


Help with concepts, advice on success, 


i 2 
informationomerades Meet with your Professor (Have you?) 


Help understanding content fora Attend Tutoring or 
specific class Supplemental Instruction 
Help with study skills, time management, etc. Take a Study Skills Workshop 


However, most students reported not receiving the messages, not reading the messages 
(if they recalled receiving them), or not taking any action in response to them (if they read 
them). For example, when asked about the messages that MCCC sent, one student said, 
“There was some. I don’t open them.” 


Messages’ effectiveness depended on who they were coming from, the language 
they contained, and the relevance of their content. 


Interviews with advisors and students suggest that colleges could be more strategic in 
crafting and disseminating messages. One important factor to consider is who sends the 
message. Some students shared that whether they decided to read an email depended on 
the identity of the sender. When asked if he read messages from the university, one Fresno 
State student said, “not all of them,” adding that he read messages that came from his 
teachers or fellow students. 


As part of their advising redesigns, Fresno State and MCCC introduced new staff 
positions—peer mentors and a retention specialist, respectively. Students’ confusion 
over the roles of these new staff members contributed to their limited engagement 
with communications from these individuals. One Fresno State student recalled seeing 
emails from a person identified as a peer mentor but not responding because he believed 
that peer mentors were intended for “people who need... help,” when peer mentors 
were part of the university’s strategy for proactively reaching all students. To help 
students understand the purpose of the peer mentors, the university sent a message 
introducing students to their assigned peer mentor in the second semester of the 
redesign. According to the peer mentor supervisor, this message made it easier for the 
peer mentors to connect with students, in part by allowing the peer mentors to refer to 
the introductory email during follow-up interactions. Similarly, the retention specialist 
at MCCC noted that students who were directly referred to her by an advisor—and 
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presumably had received some preliminary explanation from the advisor about the role 
of the retention specialist—were mote likely to respond to communications and schedule 
an appointment. These accounts suggest that students may be more receptive to messages 
that come from someone they recognize, such an assigned advisor or instructor, and that 
if messages must come from someone with whom the student is unfamiliar, strategic 
communication introducing that person can be helpful. 


Using uninviting language may also decrease student engagement with informational 
communications. MCCC advisors noted that the language used in informational 
communications may have had elements that deterred students from opening the messages 
or responding. Most notably, messages sent from MCCC’s learning management system 
automatically inserted a “Do Not Reply” precursor to the subject. One MCCC advisor 
described this phrase as “a big block” from a student’s point of view. The advisor said: 


When [the student] clicked on it, sure enough, there was the information that 
we had sent out. But I think the fact that the first thing they see in bold black is 
“do not reply’— you know, I think it’s kind of a turnoff. 


Input from marketing or communications departments and students may help advising 
staff to craft messages that are more appealing to students. 


Students indicated that they were less inclined to read messages that they felt contained 
content that was not relevant to their personal circumstances. One MCCC student said that 
he does not read announcements from the college “just because I know they won't relate 

to me.” Similarly, another MCCC student indicated that he did not read informational 
messages “unless it’s pertaining directly to me,” further underscoring the importance of 
message content aligning with students’ needs. 


MCCC advisors and project leads also suggested that the general nature of the email content 
may not have resonated with all students. Messages covered a wide range of topics, from 
childcare and transportation services to descriptions of the roles and responsibilities of 
institutional staff, such as the bursar. Colleges may improve their messages’ effectiveness 
by better tailoring their content to students’ circumstances—for instance, targeting 
information about childcare services toward students with young dependents or 
information about transportation services toward commuter students. 


Creating new staff roles can improve colleges’ capacity to communicate with 
students, but coordination between advisors and other support providers is critical. 


The addition of peer mentors to Fresno State’s student support staff increased the college’s 
capacity to communicate with students about advising, and advisors appreciated that the 
peer mentors managed reminders to students about advising appointments. Advisors 

could see that students were getting reminder calls because the peer mentors would log this 
information in the student management system. From one advisor’s perspective, “the regular 
phone calls made a huge difference” for students. The advisor noted that the phone calls were 
more personal than emails, so they helped create a sense of personalized support. 


However, while advisors knew that the peer mentors were calling students in the program 
group, there was limited communication or coordination between peer mentors and 
advisors. Advisors did not consistently check what guidance peer mentors had offered 
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students and so could not build on that information during advising sessions. In turn, 
peer mentors indicated they did not reach out to advisors to notify them of students 
experiencing more complex challenges that could warrant further intervention. While 
the college had intended this to be one of their functions, peer mentors generally felt 
uncomfortable engaging with students on topics beyond administrative matters, such 
as appointment reminders. Fresno State provided peer mentors more training in the 
second semester of the redesign, which both project leads and peer mentors noted 
helped peer mentors have more fruitful conversations with students. For example, 
one project lead described showing the peer mentors how to pace outreach efforts and 
“leave different types of messages,” which helped ease the peer mentor’s hesitation to 
repeatedly call students. 


Similarly, at MCCC, advisors did not regularly interact with the newly hired retention 
specialist. Several advisors indicated that they were not aware of the retention specialist; 
others said that though they were aware of this role, they did not know how to submit 

a referral. Those who did make referrals did not know of any mechanism in place for 

the retention specialist to “close the loop” and inform advisors about engagement 

with students who were referred. The retention specialist was hired during the second 
semester of the redesign; at the time of our data collection, the college was working 
through strategies for improving this position’s integration into existing support services. 


LESSONS FOR OTHER COLLEGES 


Expanding Informational Communication 


e Consider ways to make general information as personalized and relevant to students as 
possible. 

e Use inviting language in email communications. Consider asking marketing and 
communications departments for support in crafting messages. 

e Send students important information in an email from someone they know, such as an 
assigned advisor or faculty member. 

e lIfamessage should come from someone with whom students are unfamiliar, first send an 
introduction explaining the role of the unfamiliar staff member. 

e Leverage peers and other staff to augment support services’ capacity. Make sure to establish 
clear protocols for coordination among staff, including “closing information loops.” 


Identifying and Supporting Struggling Students 


As described in detail in Table 3, all three institutions in our study adopted strategies 
to proactively identify students in need of help and provide them with targeted, 
personalized support. First, each college sent out a student self-analysis survey at the 
beginning of the semester to give students an opportunity to report any concerns or 
needs. Advisors at UNC Charlotte and MCCC and peer mentors at Fresno State were 
responsible for receiving students’ survey responses and reaching out to students 

if their needs could be addressed immediately and/or referring to their survey 
responses during an advising session that semester. 
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Second, the institutions systematically used early-alert systems where faculty could flag 
students who were struggling academically early in the semester. During the fourth or 
fifth week of the semester, the institutions sent an early-alert survey to faculty asking them 
to identify students exhibiting signs that they needed support, such as poor academic 
performance or attendance issues. Faculty could respond to the survey or access the 
early-alert system directly at any point during the semester to flag students in need of help. 


Finally, UNC Charlotte and MCCC provided advisors with student risk scores generated 
by predictive analytic systems. These scores were intended to further inform advisors of 
students who could benefit from additional support. 


Table 3. 
Summary of Strategies to Identify and Support Struggling Students 


FRESNO STATE 


e Project leads sent a survey asking 


i Lefote] 


e Project leads sent a survey asking 


UNC CHARLOTTE 


e Project leads sent a survey asking 


students in the program group about 
their academic and advising experi- 
ences at the start of the semester. 


Faculty received early-alert surveys 

in Weeks 5 and 10 of the semester 
asking them to report on the academic 
progress of program group students. 
Faculty received an early-alert survey 


in Week 7 of the semester asking 
them to report on the progress of all 


students in the program group about 
their perceptions of the upcoming 
semester (such which course they 
were most concerned about passing). 


Faculty received an early-alert survey 
in Week 4 of the semester asking them 
to report on the academic progress of 
program group students. 


Faculty received an early-alert survey 
in Week 7 of the semester asking 


students in the program group about 
their academic experiences at the 
start of the semester. 


Faculty received early-alert surveys in 
Week 4 asking them to report on the 
progress of all students. 


Faculty submitted midterm grades 

for all students, and advisors emailed 
program group students who received 
flags and/or D or F midterm grades. 


students. them to report on the progress of all 


e Peer mentors called program group students. 


students every three weeks and, when e Advisors emailed and/or called 

applicable, asked about students’ program group students who 

early-alert flags. responded to the survey or received 
an early-alert flag. 


Identifying struggling students using a student survey was more challenging than 
anticipated. However, when available, survey data were informative. 


Atall three institutions, response rates to the student self-analysis surveys were low. 

At UNC Charlotte and Fresno State, project leads administered the survey via email 

to students and directly received the responses, which they then disseminated to the 
appropriate advisors. MCCC’s advisors administered the surveys and received responses 
directly. Project leads and advisors reported receiving very few responses. An advisor at 
MCCC recalled, “Some people only had one response. I think I had five or six.” 


Advisors offered various perspectives on why students did not respond to the survey. One 
MCCC advisor indicated finding at least one of the questions confusing and believed that 
may have contributed to the low response rate. An advisor at UNC Charlotte reported 
thinking that the survey was not appropriate for all students, explaining: 


That’s something that could be appropriate for our first-semester student. 
But a second- or third-semester—some of the questions that were being 
addressed, I feel like a student may look at those and be like... “I’ve already 
talked about those things with my advisor last semester.” 
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When students were asked about the self-analysis survey, very few remembered receiving 
it, and some confused it with other communications, surveys, and early-alert flags. When 
asked if he recalled receiving a survey, a student at MCCC responded, “I think I did get that.” 
However, when probed to see ifhe completed the survey, the student said, “I’m not really 
sure. ... [My advisor] would know better about that than I would.” 


Low response rates notwithstanding, peer mentors and advisors reported that 
information collected from the self-analysis surveys was highly valuable and allowed 
them to offer more personalized and timely support for students who did complete it. 
An MCCC advisor recalled, “I had a few [students] who seemed like they needed some 
additional support, so based on their responses, I reached out to them and gave them 
some folks that they could contact.” 


Once advisors were provided with the data from the surveys, they referred to the 
responses during advising sessions. One Fresno State advisor said: 


I felt like [the survey] was helpful because I did go in there, and I printed them 
out ahead of time. So if a student had completed one, I made sure to have that 
in front of me, and those [responses] were some of the things I did address 
with them. 


Another Fresno State advisor referred to the survey responses as a “jumping-off point” 
for the advising session. An MCCC advisor noted that a couple of students stated on the 
survey that they were uncertain if they had what it takes to be a successful student. The 
advisor said it was helpful to talk through those concerns with students. 


Though few students completed the self-analysis surveys, our interview data indicate that 
self-reported concerns from students may be highly valuable to advisors—particularly 

at institutions where time and resources are scarce. When students report issues, staff 
can focus less on identifying students who need support and more on offering the 
necessary support. Colleges might consider ways to motivate students to report their 
concerns as early in the term as possible, so advisors can intervene with students before 
the registration rush and before issues take a toll on students’ academic performance. 
Integrating surveys into processes that students already must complete, such as 
registration, may increase response rates. By connecting with students who report 
challenges early in their first semester, advisors can also establish trust and rapport with 
students so that students feel confident coming to them with issues that may arise later in 
the semester or in their college experience. 


The share of faculty who used early-alert systems varied across institutions, 
depending on institutional context. 


Faculty response rates to early-alert surveys varied across the three institutions in our 
study. MCCC and UNC Charlotte reported 70% and 65% response rates, respectively, 
across two semesters of the study, while Fresno State reported closer toa 40% response 
rate. Project leads offered some insights about the early-alert trends at their institutions. 
For instance, MCCC faculty have long been expected to use the early-alert system, which 
may explain their higher response rates. In contrast, early-alert engagement among faculty 
has historically been low at Fresno State, and even though the faculty response rate 
following the college’s advising redesign seems low, it is still higher than previous faculty 
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response rates. Fresno State’s project leads suggested this increase may be attributed to 
communications to faculty prior to the study period to encourage the use of early alerts. At 
UNC Charlotte, a technology change between the first and second semesters of the study 
may have slightly depressed the use of the early-alert system by faculty, as they were likely 
still getting familiar with the new system in the second semester. Moreover, the new system 
did not include a feature that enabled faculty to give students positive feedback, and project 
leads believed that faculty may have been using that feature prior to the change. 


Strategies to get faculty to participate in the early-alert process are critical for early-alert 
systems to function optimally, since faculty possess vital information about student 
performance that can be shared with advisors so that they can collaboratively support 
students in need. Additionally, faculty make up a large stakeholder group at colleges, 

and they interact with students most frequently, so they can provide critical insights 
about how to communicate with students and how to provide support when students are 
struggling. Furthermore, faculty input into the development of early-alert messages could 
help them understand what messages their students will receive and prepare students 
accordingly. Ensuring that faculty are aware of the interventions that follow as a result of 
the alerts may also lead to their increased use of the alert systems. 


Early-alert flags did not always tell advisors what they needed to know to 
intervene with students effectively. 


When faculty raised flags about student performance, they often did not include specific 
information about what triggered the alert. In these cases, it was difficult for advisors to 
make pertinent recommendations and offer guidance to students. Students could also be 
affected by the lack of detail in the early alerts. One MCCC advisor said, “A lot of times, the 
students have no idea why they have that early alert, so they need to contact the instructor 
and find out exactly what it is.” Colleges should look for ways to make sure the flags in their 
systems clearly communicate the nature of the issue that requires an intervention. Colleges 
can also encourage faculty to elaborate on the issue when submitting an early alert. 


With limited time and resources, it was challenging for advisors to provide 
personalized attention to students who were flagged in early-alert systems. 


Advising technologies made some aspects of advisors’ work more efficient, such as 
identifying struggling students using early-alert data. However, advisors’ limited capacity 
was a barrier to personalizing their responses to students flagged in early-alert systems. 


Advisors reported encountering new and time-consuming technological and data 
management tasks as a result of the redesign. One MCCC advisor explained that even with 
an early-alert system to send messages to flagged students, “still, it takes time because you 
have to go in, you have to identify [flagged students], then you have to set up your mail 
merge and get that all straightened out.” At UNC Charlotte, the project lead managed 
most of these administrative tasks, including setting up mail merge parameters for the 
advisors. This process made initial messages easier to disseminate, but advisors needed to 
manage electronic or in-person follow-ups, which took considerable time and effort. One 
UNC Charlotte advisor described this work, stating, “I’ll havea student come in and be 
like, ‘Hey, you sent me an email saying I [am] at risk in these classes,’ and I’m like, ‘I did?’ 
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And then I am going back and checking because I just have so many students that it is just 
hard to manage all of them like that.” 


Some advisors mentioned that they had to be selective in which students they reached out 
to regarding flags. At least one advisor indicated that she was not able to provide students 
with positive feedback or intervene with students who only received one flag because she 
only had time to respond to the highest-need students. One UNC Charlotte advisor said: 


We get so many flags. Some mornings, I look at my email, and I may have, 

I don’t know, 10 or 15 ...different flags for different things. Some of them 
are [ for positive feedback], and some of them are not. Some of them are a big 
deal, and some of them are not.... Like I said, if there are multiple flags for 
one student, that’s something that would kind of catch my attention. If it’s 
just one thing, I may not reach out. 


To help advisors use their limited time efficiently, key project staff at each institution 
drafted customizable templates for advisors to use to respond to early alerts. However, this 
approach presented an issue for advisors who had developed relationships with students 
and did not want to seem cold or impersonal in these communications. 


Advisors expressed concerns about the language used in automated responses 
sent to students via early-alert systems. 


Advisors across all three institutions reported being worried about how alert messages, 
particularly those indicating risk, were framed for students. One Fresno State advisor 
described the messages as potentially “scary,” especially for first-year and first-generation 
students. A UNC Charlotte advisor discussed the importance of personalizing the 
messages sent to students as a follow-up to faculty-triggered early alerts “so that they 
know that we are paying attention to what’s going on with them.” The advisor described 
the importance of using a “softer tone” without abandoning a “sense of urgency.” 


When designing and implementing an early-alert system, colleges should consider 
gathering wide-ranging input from advisors, faculty, and students to design messaging 
strategies that may be more likely to be welcomed and acted upon by students. 


Students had varied reactions to early-alert messages, ranging from fear to 
indifference to appreciation. 


Students who received warning messages about academic troubles reported a range of 
feelings. Some students responded positively to receiving warnings about academic 
performance. One UNC Charlotte student said the warning messages made him feel like 
“okay, they really care about what I’m doing. They want me to go ahead and get my stuff 
together. [They] don’t want me to fail. So, then I immediately set up an appointment.” 
When asked how she reacted when she received an early alert, another UNC Charlotte 
student responded, “I started working harder and studying more for tests.” A third UNC 
Charlotte student said that receiving an early alert made her realize her instructor was 
paying attention to her individual performance: 


It’s kind of easy to feel like when you're in these big classes, that the professor 
doesn’t notice me specifically. I’m just another name. [Receiving an alert is] 
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like, “No, I know you, I know you're not doing great in this class, and you 
should step up your game.” I think it’s more encouraging, honestly. It’s more 
like this person’s putting all this effort into teaching me and letting me know 
her office hours and letting me know how to get help.... And she’s concerned, 
and I’m concerned too, like I should actually reach out and try and get help. 


At MCCCand UNC Charlotte,’ students could receive positive messages (kudos) as well as 
those expressing concern. Students who received positive messages appreciated receiving 
them but sometimes noted that they were not personalized. One MCCC student said: 


I don’t know what the message was, but it was pretty generic. It wasn’t like a 
personalized message or anything like that. It was encouraging. I know that it 
wasn't, like, typed out, written all nicely, and that good stuff. But it’s still nice 
to see someone cares about the progress, at least. 


One UNC Charlotte student reported receiving a message that said, “Keep up the 

outstanding work.” Another UNC Charlotte student referred to the positive messages 
as receiving a “gold star” and as “pat-on-the-back emails.” The student appreciated the 
message, adding that it felt good to know that the professor recognized his hard work. 


Not all students thought early alerts were a good idea, however. One UNC Charlotte 
student said, “I think students know what they are doing. I don’t think they need, like, an 
email saying they are doing bad. Because they pretty much know already.” Other students 
described early alerts as “red flags” or “dings.” When asked whether she should have a 
received a warning, another UNC Charlotte student responded: 


Yes and no. Because I transferred into the class—like, I switched into it late—I 
had missed the first assignment, so I had a zero for it....So then Ihad to talk 
to [the professor], and I got it figured out. So, I knew what [the alert] was for, 
but it was kind of annoying. Nobody wants a red flag. 


Some students indicated they did not always know from whom the messages originated 
or why they were receiving them. It is also possible that some students viewed the 

early alert as an indicator that they were past the point of help. A student who dropped 

a course after receiving a flag said, “I knew the flag was probably coming, but I guess it 

is kind of discouraging to see that. It isa good thing, though, definitely, to be aware of 
your progress.” When asked if she dropped the course because of the flag, the student 
responded, “No, I guess I kind of already knew I was going to have to withdraw based 
on how much higher I would need to raise my grade.” A student who was failing a 

class described how he felt when he received the early alert: “I was [made] a little bit 
uncomfortable by it because no one likes to see an F.” When asked if he talked with anyone 
about it, he responded, “No, because there’s nothing you can do. Like, you have an F.” 


One strategy to address student confusion about early-alert messages may be to develop 
some boilerplate language about early alerts for faculty to include in their syllabi. 
Informing students about early-alert systems could reduce confusion and help students be 
more responsive because they would know what it means to receive that type of message. 
In explanations of early-alert systems and in the language used in the alerts, colleges 
should frame the message in a way that is supportive for struggling students. Students 
may be more receptive to receiving alerts if they believe that the professor is expressing 
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sincere concern. Colleges should also consider including opportunities for faculty to 
provide positive feedback to students. Positive messages can be meaningful for students, 
particularly those who have struggled in the past or lack confidence as college students. 


Advisors expressed concerns about using predictive analytic data to identify 
and support struggling students, underscoring the critical role of the advisor in 
interpreting these data. 


In addition to student survey responses and early-alert data, UNC Charlotte and MCCC 
provided advisors with student risk scores generated by predictive analytic systems. 
Advisors at both institutions expressed concerns about using these data to inform their 
engagement with students. A technological glitch at UNC Charlotte resulted in the 

risk scores of a large subset of the university’s population not being updated when new 
information about students’ academic performance during the fall term was available. One 
UNC Charlotte administrator noted that “skepticism about data quality” made advisors, 
who had already expressed concerns about the tool, even less inclined to use it. She said 
advisors were worried that predictive analytics were “guiding people out of majors where 
they could have been successful too soon,” adding, “I think there was concern about 

the use of the predictive analytics and turning this into a system where people might be 
stereotyped or categorized unfairly.” In response, UNC Charlotte reduced its emphasis on 
predictive analytics after the first semester of the study. 


Similarly, MCCC advisors reported that the predictive analytic data were not always 
actionable. Some students identified as at-risk were designated as such due to circumstances, 
suchas the location of their home, that advisors could not do anything to address. MCCC 
responded to advisors’ concerns by implementing a new model for assessing which students 
would be required to meet with an advisor for the second semester of the project. In this 
model, predictive analytic data were still used as one way to identify a larger subset of 
students for extra support, but indicators over the course of the semester, such as grades, 
were used to determine which students needed an additional advising session. 


These observations illustrate the limitations of predictive analytic tools as currently 
used and underscore the fact that human judgment is necessary to properly make use of 
the data. Predictive analytics may help colleges and advisors identify broad categories of 
students who may benefit from support, but they cannot effectively diagnose specific 
struggles a student may be having. 
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LESSONS FOR OTHER COLLEGES 


Identifying and Supporting Struggling Students 


e Beattentive to the language and tone used in student surveys and early-alert messages. 

e Engage students in the development of student surveys and early-alert messages. Doing so 
may result in content that is more likely to elicit a response. 

e Make it easy for students to report challenges they are experiencing by integrating surveys 
into processes that students are already completing, such as registration. 

e Coordinate with faculty in developing early-alert surveys and messages, and educate them 
about the interventions that will be implemented in response to early-alert data so that they 
are aware of the implications of flagging a student. 

e Customize early-alert surveys so that flags from faculty inform advisors about the type of 
challenges a student is experiencing, so they can better offer support. 

e Utilize predictive analytic data to identify students who may need support, but recognize the 
limitations of predictive analytics’ ability to inform specific interventions. 

e Set aside time in advisors’ schedules for advisors to review survey, early-alert, and predictive 
analytic data and implement interventions in response to these data. 


Improving Advising Sessions 


The final component of the advising redesign was a required advising appointment for 

at least a subset of the students targeted by the project. (See Table 4 for details.) Colleges 
placed registration holds on these students to compel them to meet with an advisor.* The 
advising sessions were intended to provide targeted academic and nonacademic support 
and help students clarify their academic and postgraduation pathways. To support advisors 
in facilitating these discussions, colleges developed and promoted the use of an advisor 
toolbox. Advisors also had access to early-alert and predictive analytic data, which they 
could use to guide their discussions with students. 


Table 4. 
Summary of Strategies to Improve Advising Sessions 
FRESNO STATE iW fefete] UNC CHARLOTTE 
e Program group students were required e Program group students were required e Program group students who received 
to meet with an advisor for a one-hour to meet with an advisor at least once Dor F midterm grades were required 
advising session each semester of the per semester of the study. to meet with an advisor. 
study. e The advising toolbox emphasized e The advising toolbox highlighted three 
e The advising toolbox emphasized utilizing technology (MyCareerPlan) main questions to ask about students’ 
utilizing technology (MyDegreePlan) for career planning discussions with academic experiences. 
for degree planning with students and students and instructed advisors to 
provided a GradesFirst template for determine if students were experi- 
advising notes. encing any of nine listed academic and 


nonacademic issues. 


e The toolbox guided advisors to take 
notes using Starfish’s SpeedNotes 
feature (a checklist aligned with the 
toolbox topics). 
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Registration holds got students to see their advisors, but the circumstances and 
timing were not ideal, as many students put off advising sessions until absolutely 
necessary for course registration. 


At Fresno State, although registration holds were placed at the start of the semester, most 
students waited until peak registration periods for the following semester to meet with an 
advisor and have the hold removed. Asa result, the academic support and planning that 
these advising sessions were meant to provide, which would have been most beneficial 
early in the semester, did not take place until midway through the semester. Advisors 
found it challenging to make appointments with all of the students with registration 
holds while also meeting the needs of other students looking for guidance during the 
registration period. One Fresno State advisor had blocked off 45 hours of his schedule 
earlier in the semester to meet with students with holds but found that “very few students 
[were] responding, soa lot of that time ended up going unfilled.” 


The registration hold system at MCCC was especially weak as a tool for motivating students 
to seek advising—possibly, MCCC project leads and advisors suggested, because the college 
has flexible registration periods, and some courses do not reach capacity enrollment. In this 
context, students with registration holds have little incentive to see an advisor before the 
start of the following semester to enroll in courses. One of the MCCC project leads explained 
that “students just kind of wander in whenever to get registered.” Requiring them to come 
back, having prepared for a career planning discussion, for a scheduled appointment that 
would meet the objectives of the redesign turned out to be difficult. 


Overall, registration holds did not motivate students to see an advisor until doing so was 
absolutely necessary to register for courses for the next semester. As a result, any guidance 
that advisors wanted to offer students during advising sessions to improve their course 
performance did not reach students until at best halfway through the semester. Alternatives 
to registration holds, such as offers of early registration or tickets to institution-sponsored 
events, may work better to incentivize students to meet with an advisor early in the 
semester. Students may also be more inclined to seek out advising throughout the semester 
if they feel a stronger connection with their advisor, so it may help to take steps to strengthen 
the advisor—advisee relationship, such as making it clear that each student is assigned a 
specific advisor who is there to provide individualized support. 


Software and online tools helped advisors streamline appointment scheduling, 
prepare for advising sessions, and take notes during sessions. However, some 
structural changes may be necessary to ensure that advisors can make the best 
use of them. 


To help students schedule appointments with their advisors, institutions used an email 
campaign system that allowed advisors to send students a message with a link to a page where 
they could book an appointment. Advisors and students liked this system and reported that 
the scheduling feature was easy to use. One UNC Charlotte advisor said, “Students appreciate 
that ability to immediately see that availability and to make an appointment.” 


In interviews, advisors identified the tools that were most helpful in preparing for advising 
sessions. For example, during the advising redesign, UNC Charlotte launched a software 
application that allowed advisors to see multiple types of information about students, 
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including their schedules, in a single window. An advisor explained that the application 
“brings what we are using multiple technologies for now into one place,” making it easier 
to review more information about the student and ask more targeted questions. (See 
Figure 2 fora representation of this application.) 


Figure 2. 
Representation of UNC Charlotte’s Student Information Application 
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Institutions also deployed tools to make note-taking during advising sessions a more 
common and consistent practice. At UNC Charlotte, project leads created a checklist of 
topics that should be discussed in advising sessions. In addition to any information advisors 
chose to document about a session, advisors marked off items that were discussed on the 
checklist. Advisors noted that the checklist served as a reminder to “double-check what we 
touch base on” during the advising sessions. As part of its advising toolbox, Fresno State 
created a template with fields where advisors could input information related to a student’s 
coursework, time management, study skills, major, and degree plans. While the template 
was not embedded in the college’s advising technology, all advisors copied and pasted the 
completed templates into their shared system so that they were easily visible for all advisors 
who may need to access a given student’s profile. The template provided a structure that 
motivated advisors to write notes that, as one Fresno State advisor said, give “a much clearer 
picture of what was discussed” in an advising session compared with the “very generic, 
brief notes” that were common practice before the redesign. Another Fresno State advisor 
described filling in the template during advising sessions rather than writing notes from 
memory after the session was complete, as she had done previously. 


Advisors’ use of technologies was influenced by the organizational structure of advising 
at each college. For example, while UNC Charlotte’s communication tool allowed advisors 
to generate appointment sign-up campaigns, some departments primarily offered walk-in 
sessions and preferred not to shift to an appointment structure. While advisors from 
these departments liked the campaign tool, they did not use this functionality. Even 
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user-friendly tools may not be utilized if the organizational structure does not make their 
use feasible or relevant. In selecting tools, colleges should consider their functionalities 
in the context of existing structures and, if there is misalignment, determine whether 
structural changes are warranted or if another tool would be more appropriate. 


Students and advisors described technology-based degree planning discussions 
as time-consuming but useful. 


Fresno State and MCCC utilized degree planning technologies to enable students to map 

out the courses they need to take each semester to complete a degree. Degree planning tools 
make it visually easier and more efficient to build a multisemester plan and compare and 
contrast pathways for different programs of study. At both institutions, students generally 
found value in the technology-based degree planning process, with one Fresno State student 
describing it as “extraordinarily helpful.” Still, both advisors and students reported that 
students had a strong preference for developing plans, at least initially, with support from an 
advisor. One Fresno State student reported that an advisor’s feedback helped him understand 
the difference between two math courses that both fulfilled a requirement. With the 
advisor’s help, he said, he picked the one that was more appropriate for his interests and skill 
set. The student said, “It was just really reassuring for that kind of instance” to hear from the 
advisor, explaining that the advisor knows “classes I can sign up for or what’s unnecessary.” 


Students’ preference for direct support from an advisor in developing their degree plans 
meant that advisors needed to allocate more time than originally anticipated during 
advising sessions for this activity. As one Fresno State advisor noted, the advising session 
needed to include enough time to teach students how to use the degree planning tool 

as wellas how to build a program plan using the tool. Fresno State extended advising 
appointments to a minimum of 45 minutes for program group students, making these 
time-intensive discussions possible. For colleges that do not have the capacity to extend 
advising appointments, finding ways to get students in during non-peak periods and 
streamline administrative advising tasks, such as course registration, can allow more time 
for degree exploration and planning discussions. 


MCCC students and advisors highlighted both benefits and limitations of career 
planning technologies, with such tools’ helpfulness varying according to students’ 
age and tenure at the college, among other factors. 


MCCC used a career planning tool, which included four career assessments and other career 
exploration functionalities, to allow students to explore their interests and see how they align 
with career options. Advisors found the data from these assessments useful for supporting 
students in identifying a career path that would bea good fit. One MCCC advisor, for example, 
described how she used the results of the career assessment to help a student understand 

how her intended degree and career pathway might not align well with her strengths. The 
student worked in a dental office and was planning to pursue a career as a dental hygienist. 
The advisor pointed out to the student that “dental hygiene wasn’t even on the list” of 

career options recommended by the career assessment tool, nor were related majors, such 

as physical or occupational therapy. To further underline her point, the advisor searched for 
dental hygiene in the career planning tool, which showed it was a weak match for the student. 
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The advisor said it “was a valuable discussion to have” with the student. Another advisor 
similarly described how he looked at the career assessments with a student “and really dug into 
it.” Through this exercise, the student, who was “questioning what she was doing,” received 
feedback that helped her decide to pursue a certificate in payroll management or bookkeeping. 


Some MCCC students also pointed to the utility of the tool in helping them understand 
different career opportunities, especially when accompanied by support from an advisor. One 
student described how an advisor “discussed how to navigate” the tool and “how to search 
for specific careers.” He also was able to find information about pursuing careers in different 
states, such as each state’s requirements and median salary. From this conversation, the 
student said, he was able to “basically get an idea of ... what is it that fits career-wise.” 


While the career assessments were helpful for some students, others felt that they were not 
aligned with their needs or life circumstances. For example, one MCCC student, who was 
planning to pursue a business economics degree, said that his advisor had used the career 
assessment data to suggest that artistic careers might be more appropriate. The student, 
however, felt that he had to consider his parents’ preferences and the financial prospects of 
the career path he pursued. “I definitely appreciate the advice, and I would take it ifI were in 
different circumstances,” he said. “I don’t think I would have chosen business administration 
if] had full control. But, you know... it’s one of those difficult things.” 


Advisors and students at MCCC also noted that the career planning tool did not do much 
to enhance discussions with students who were nearing graduation or older students. One 
older student said, “You know, at 83 [years old] .. . career? I’ll leave that for the kids.” This 
student elected to skip the required advising appointment and did not complete the career 
assessment. Similarly, one advisor said, “Somebody who’s graduating, or a 67- or 64-year- 
old, they don’t need the intrusiveness of what I’m providing at this point.” Students who 
were nearing graduation or older were generally clear about their career intentions. Asking 
these students to complete a career plan and meet with an advisor to discuss the plan was a 
misallocation of resources from this advisor’s perspective. 


Because not all students need intensive career planning, differentiating policies and 
practices regarding career planning for different types of students should help ensure that 
advising resources are allocated efficiently and that students who are likely to benefit from 
career support are directed to relevant resources. 


Some advisors found the advising toolbox useful, and their reactions to it suggest it 
may have utility for professional development. 


In general, Fresno State advisors perceived the toolbox as useful because it gave them direction 
and helped them enhance their interactions with students during advising sessions. (See Figure 
3 for an excerpt from the toolbox.) As one Fresno State advisor said, the toolbox provided “tips 
and pointers that I could think about as the conversation was going on.” Advisors typically 

did not feel compelled to address every topic or use every question listed in the toolbox. “If 

it flowed into those questions, it flowed, and if it didn’t, it didn’t,” the same advisor added. 
Another advisor said the toolbox “laid out what we were wanting to do” to ensure effective 
advising sessions, while others said that it gave the advisors clear guideposts to work toward. 


The suggested questions for students included in the Fresno State toolbox also helped advisors 
reflect on their practice and identify areas for potential development. For example, one advisor 
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described reflecting on advising sessions and her use of the toolbox. She said, “There [were] a 
few times where I was like, ‘I should have asked that question. That would have been a better 
leading question or a better detailed question to ask in that area.” 


Project leaders at Fresno State and UNC Charlotte described interest in continuing to use 
the toolbox both in their advising practice and by advisors institution-wide. During an 
interview in the final semester of data collection, one Fresno State project lead who also 
advises students said, “The one thing I will say that I truly took away from this... would 
be that toolkit. ... There are questions on there that now I’m using all the time because I 
never thought of asking it that way.” UNC Charlotte’s project leads also expressed interest 
in integrating the toolbox into onboarding practices for new advisors and professional 
development for advisors with a limited background in counseling. 


The concept of a toolbox received the least traction among advisors at MCCC, seemingly 
because they found it redundant with the work they were already doing. Most MCCC 
advisors described the toolbox as “just stuffI do with every single student anyway,” 
though they did mention finding some features useful. MCCC advisors said that 

having the toolbox, especially the list of nine issue areas that may be affecting student 
performance, was a useful reminder, particularly for new advisors. 


Figure 3. 
Excerpt From Fresno State’s Toolbox 


OBJECTIVES BY TOPIC GUIDING QUESTIONS 


MyDegreePlan 


e Confirm student has completed MyDegreePlan that extends e Why do you think this plan makes sense to you? 


beyond one semester. 


Review plan to identify and help student understand potential 
pitfalls in what student has planned. 


Help student craft and/or adjust as necessary a 
MyDegreePlan based on assessment of aspirations, interests, 
and values. 

Teach student to create a “game plan” or develop strategies 
for next semester that will help him/her achieve academic 
and career success. 


What part of this plan might be challenging? 


Can we think of strategies to address those challenges in 
advance, so you have a game plan? 


Academic and Career Goals 


Address student concerns pertaining to major selection. 
Support student understanding of degree requirements for 
major(s) and/or program of study. 

Help student understand and articulate his/her career 
options and opportunities based on major selection. 

Check on how student is performing in major classes to 
inform how to guide student on success in their chosen major. 
Direct student toward activities and habits that will nurture 
and support his/her academic and career goals. 

Refer to University Advising Center for major exploration or 
Career Development Center for further career guidance, as 
needed. 

Refer students to other campus involvement opportunities 
such as clubs, study abroad, internships, research 
opportunities, etc. 

Guide student to understand connection between 
current-term performance and long-term goals. 


How do you feel about the major you have selected? 

What about this major feels like it is a good fit? 

What isn’t a good fit? 

What questions/concerns do you have regarding your major? 
What type of career do you hope to gain from earning your 
degree in this major? 

What jobs are most interesting to you? What type of work are 
you interested in doing? 


What internships have you considered? What type of 
internships are you interested in doing? 

What other activities, like study abroad or research with a 
professor, are you interested in exploring? 


What impact do you think your current semester will have on 
your college degree? Your life after college? 
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Figure 4. 
Core Issues Outlined in MCCC’s Toolbox 


ISSUE AREAS yXegule) Tag sit) 


1. 


Career indecision 
When a student is uncertain as to his or her successful 
career path 


Referral to Career Services 
To-do to further explore MyCareerPlan independently 


. College readiness 


When a student isn’t academically prepared to be successful 
in college-level coursework (study habits, etc.) 


Encouragement to take Strategies for College Success 101 if 
not already taken 


Referral to Tutoring if appropriate 


. Educational value 


When a student doesn’t understand the intrinsic value 
of earning an education, making him or her question the 
purpose of this experience 


To-do to further explore MyCareerPlan independently 


. Health issues 


When a student has diagnosed or undiagnosed symptoms 
of anxiety, depression, or other (mental or physical) illnesses 
that interfere with his or her academic success 


Referral to Student Support and Referral Team 


. Interference factor 


When a student has nonacademic issues that are interfering 
with academic success 


Referral to Student Support and Referral Team or Financial 
Aid (or additional cohort program depending upon student 
need/interest-Minority Student Mentoring Initiative, Keystone 
Education Yields Success, Act 101 Scholars Program, etc.) 


6. Mismatch between skills, interests, and abilities 


When a student has an educational goal that doesn’t align 
with his or her actual skill level or abilities 


Referral to Career Services 
Referral to Disabilities Services (if applicable) 


. Self-efficacy 


When a student doesn't believe that he or she can actually 
succeed in school (lacks the Little Engine’s mentality) 


Referral to the Director of Student Retention 


. Sense of belonging 


When a student does not feel a sense of belonging or 
affiliation with the institution 


To-do encouragement to explore campus involvement 


. Transition management 


When a student is having trouble adopting an identity as 
a college student, transitioning from other roles as a high 
school student or worker 


Referral to the Director of Student Retention 


Colleges looking to support advisors in their professional growth and to establish norms 
for high-quality advising that are consistent across the institution may consider creating 
an advising toolbox. Our interviews, especially with advisors at Fresno State and UNC 
Charlotte, indicate that engaging advisors in using a toolbox may help them to reflect on 
their practice and identify areas for development, pointing to the potential for a toolbox 
to promote advisors’ professional development and growth. 


That said, an advising toolbox is only useful if advisors are receptive to using it. Our 
data point to important lessons learned about the process of developing a toolbox 

that have implications for advisors’ willingness to use it. In our study, advisors were 
receptive to a toolbox if it was designed to guide rather than dictate their approach with 
students. Moreover, if advisors are part of the toolbox design process, they can ensure 
that the resource aligns with what they identify as the areas of their practice that need 
enhancement, further bolstering the utility of the tool. 


Finally, the toolbox should ideally be a living document that is revisited and revised 
over time with the input of advisors. Through periodic revision processes, advisors 
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may benefit from opportunities for self-reflection and growth and stay up to date with 
changes in institutional processes, structures, and strategic goals. To facilitate this type 
of sustained engagement, colleges may consider establishing toolbox revision structures, 
such as recurring meetings to discuss and refine the toolbox. 


Some types of toolbox questions were especially useful in deepening advisor- 
student interactions. 


Advisors identified three types of guiding questions from the toolboxes that they thought 
served them well in deepening their engagement with students. First, questions probing 
the positive aspects of students’ experiences were well received. For example, one UNC 
Charlotte advisor said that before the college implemented the toolbox, he typically 
focused on investigating ifa student was struggling and did not often ask students, “What 
is it that you're doing well?” The toolbox prompted him to change his practices. Similarly, 
a Fresno State advisor described how asking positive questions—such as “What’s your 
most enjoyable course?”—helped her probe deeper into students’ academic experiences 
and better understand the alignment between students’ interests and their degree and 
career plans. 


Second, advisors found guiding questions on challenging circumstances useful for 
broaching topics that could be difficult to discuss or uncover. As one UNC Charlotte 
advisor put it, “sometimes [students who are struggling] will say everything is going well,” 
or the challenges they are facing “may not necessarily be academic.” In these cases, the 
toolbox reminds advisors to ask about things that are not going well, which often revealed 
issues “that may or may not have naturally have come out of that conversation ifI didn’t 
specifically ask it.” Similarly, another UNC Charlotte advisor described engaging witha 
student who was performing decently academically but who, when asked about things that 
were not going well, revealed nonacademic struggles. The advising session “turned into 

a conversation about the on-campus food pantry and referrals to the nutritionist.” These 
topics might not have come up without the prompt included in the toolbox. 


Finally, advisors noted that they liked questions from the toolbox that asked students to 
explore the behaviors or habits that could be affecting their academic success. For example, 
one Fresno State advisor said, “I really like asking students how they like to spend their time 
outside of school [to] reduce stress.” Another Fresno State advisor described asking about a 
student’s strategy for studying for his most challenging course and advising the student to 
find a place on campus to study to avoid the distractions he described facing at home. 


Colleges considering an advising toolbox may benefit from engaging advisors to determine 
if there are specific types of questions that they would like to include. The three types of 
questions that advisors in our study observed deepened their relationships with students 
could be a useful starting point. 
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LESSONS FOR OTHER COLLEGES 


Improving Advising Sessions 


e Consider other ways to incentivize students to meet with an advisor besides registration 
holds, such as offering early registration for those who seek advising. 

e Make use of technologies to help advisors streamline some of their activities, such as 
scheduling appointments and taking notes. Ensure structures and resources are in place to 
support advisors in using these functionalities. 

e Ensure advisors have adequate time and resources to work with students to develop 
academic plans. 

e Tailor career advising practices according to students’ needs instead of using a one-size-fits- 
all approach. 

e Create an advising toolbox by working with advisors to develop a resource that guides 
advisors in the areas where they think they may benefit from additional support. 


Conclusion 


This report describes the experiences of three colleges and universities UNC Charlotte, 
Fresno State, and MCCC— that implemented advising reforms in the spring and fall of 
2017. This report is part ofa larger portfolio of work that CCRC and MDRC are producing 
as part of our research partnership with UNC Charlotte, Fresno State, and MCCC. 
Previously published products include a framework for advising redesign (Community 
College Research Center, 2017), a report describing the initial versions of each institution’s 
reforms (Kalamkarian et al., 2018), a report on interim student outcomes (Mayer et al., 
2019), anda report on longer term student outcomes (Miller et al., 2020). 


While the advising redesigns at UNC Charlotte, Fresno State, and MCCC did not result 
in a statistically significant improvement in student outcomes (Mayer et al., 2019), the 
redesign process yielded important lessons about reforming advising practices, which 
are delineated in this report. Based on our implementation findings, we offer several 
suggestions for colleges engaging in or considering this work: 


e Acknowledge up front that technology alone cannot resolve advising and student 
support issues, nor can it replace face-to-face conversations about education and 
career planning. 


e Consider ways to encourage or make it easier for students to self-report concerns. 


e Call for input from advisors, faculty, students, and communications or marketing 
teams on the content of risk-related messages and alerts. 


e Consider alternatives to registration holds to incentivize students to meet with an 
advisor, as registration holds may have unintended consequences. 


e Provide ongoing support for staff in implementing reformed advising practices. 
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The findings and implications presented in this report align with key takeaways from 
CCRC’s broader portfolio of research on advising reform. Previous CCRC research found 
that—similar to the experiences at the three institutions included in this report—advising 
reform at other iPASS grantee institutions was a complex, multifaced, and iterative 
undertaking (see Karp et al., 2016; Klempin & Pellegrino, 2020). iPASS institutions 

are still at an early stage in implementing reforms at scale. At this point in the redesign 
process, we observe at best small improvements in institutional performance indicators, 
such as the average number of credits attempted, during the iPASS grant period, and 
even those changes cannot be attributed directly to the advising reforms (see Velasco et 
al., 2020). Still, there is qualitative evidence that iPASS grantee institutions are iterating 
toward a model of advising that theory and research suggest is more effective, especially 
when aligned with broader, whole-scale reforms that transform the student experience, 
such as guided pathways or state- or system-level reforms (Klempin & Pellegrino, 2020). 


While we have learned a great deal about advising reform, there is much more to 
understand. Subsequent analyses of student outcomes at institutions at later stages of 
advising reform can help clarify the role of advising reform in supporting improvements 
in postsecondary outcomes. Advising research that places greater focus on racial, 
socioeconomic, and gender equity can be instrumental for understanding how advising 
can support broader institutional efforts to improve equity in student outcomes. 
Moreover, further study of the changing roles, responsibilities, and functions of advisors, 
faculty, and other support providers in a technology-mediated context can further 
illuminate how technology can facilitate effective advising. 


Endnotes 


1. This count does not represent distinct individuals; a majority of advisors and project 
leads were interviewed once in spring 2017 and again in fall 2017 to understand their 
experience over the course of the redesign. 

2. Asanalysis progressed, the team adjusted and added code titles, code descriptions, and 
subcodes as needed. The final codebook included 19 main codes and 84 subcodes. (See 
Appendix Table A3 for excerpts from the codebook.) The research team met weekly to 
discuss codes, questions, and emerging themes and to update other teams on progress. 
The research team leader conducted coding checks over the course of the months-long 
analysis period. 

3. UNC Charlotte’s early-alert system included kudos during the first semester of the 
study. In summer 2017, the university moved to a system that did not include kudos. 

4. Aregistration hold prevents a student from registering for courses. The hold is lifted 
once the student fulfills the requirements specified by the hold, such as meeting with 
an academic advisor. 
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Appendix 


Table Ai. 


Advising Technologies Utilized at Each Institution 


INSTITUTION uxele) & FUNCTIONALITIES UTILIZED 
GradesFirst e Raising flags for students struggling academically 
e Sharing notes among advisors 
Fresno State U.Direct e Degree planning (referred to as MyDegreePlan) 
e Auditing courses that the student has taken to determine which courses 
U.Achieve and what number of credits align with the requirements for the student’s 
intended degree 
JobZology e Career assessment and planning (referred to as MyCareerPlan) 
eePaeh Gannsok e Scheduling advising appointments 
e Sharing notes among advisors 
Starfish Early Alert e peeing Hieee tor students struggling academically 
mccc e Reporting midterm grades 
Ellucian’s education planning e Degree planning 
Civitas Illume e Assigning students a “risk score” based on a predictive analytic algorithm 
Custom tool oreated in ee ee 
partnership with Blackboard logging onto their portal 
Starfish Early Alert e Raising flags for students struggling academically 
UNC Charlotte EAB Student Success e Assigning students a risk score based on a predictive analytic algorithm 
Collaborative e Sharing notes among advisors 
Table A2. 


Target Populations for Enhanced Advising 


INSTITUTION ay-\3\cl-am fe) 10] wale) | 


Fresno State targeted students who at the beginning of their second year had earned 15-74 credits 
and had a 2.0-2.9 GPA. The target population excluded students who already experienced proactive 


Breang State outreach based on participation in high-demand majors, such as nursing, and special programs, such as 
athletics. 
MCCC’s enhancements focused on degree-seeking continuing students with a low or moderate likelihood of 
mccc oe Z Pade : 
persisting to the next semester based on the college's predictive analytics tool. 
UNC Charlotte identified two cohorts of students for enhanced advising. The first cohort included 
continuing students who at the start of their second year had completed less than 60 credits and had 
UNC Charlotte 


a 2.0 or higher GPA but a low or moderate likelihood of graduating based on the institution’s predictive 
analytics tool. The second cohort included transfer students. 


Table A3. 
Codebook Excerpts 
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CODE DEFINITION 


Context 


Context_reforms 


Recently implemented changes or plans for upcoming changes to the structure of support 
services and related technology. Only applies to reforms that are in process or to which people 
are still adjusting. Vay be double-coded with “service structure_overview.” Does not include the 
RCT or iPASS. 


Capacity for intervention 


Capacity_staff 


Capacity_staff time 


Participant’s description of the institution’s capacity to implement the intervention in terms of 
staffing. Addresses whether the institution had sufficient staff (advisors, graduate students, etc.) 
to implement the intervention and, if not, what steps the institution needed to take to add staffing. 


Participant’s description of whether individual staff have enough time to implement the 
intervention (e.g., advisor description of how long it takes to send out messages and whether the 
advisor has sufficient time for this task). 


Outreach—Risk identification 


Identification_early alert 


Identification_midterms 


Description of what early-alert data is conveyed to or referred to the service provider and how 
this information is conveyed. Includes referrals from peer mentors or other support services 
for further intervention for flagged students. Must be double-coded with student and service 
provider type. 


Description of what midterm data is conveyed to or referred to the service provider and how 
this information is conveyed. Includes referrals from peer mentors or other support services 
for further intervention for flagged students. Must be double-coded with student and service 
provider type. 


Outreach—Risk intervention 


Intervention_early alert 


Intervention_midterms 


Description of whether and how the service provider reaches out to students flagged as at-risk 
based on early-alert data. Includes referrals of flagged students from other support staff, such 
as peer mentors. Does not include discussion of early alerts during advising sessions. Must be 
double-coded with student and service provider type. 


Description of whether and how the service provider reaches out to students flagged as at- 
risk based on midterm exam data. Does not include discussion of early alerts during advising 
sessions. Must be double-coded with student and service provider type. 


Advising session 


Session_logistics 


Session_materials 


Session_materials_toolbox 


Session_education 
planning 


Session_career planning 


Session_referrals 


Description of how advising sessions are set up, including whether they are scheduled, drop-in, 
student-initiated, advisor-initiated, etc. Includes descriptions and perceptions of session length 
(i.e., long enough or not). Advisor transcripts must be double-coded with student type. Add a 
memo if unclear about adherence to intervention or intervention design. 


Description of materials, resources, or technologies used during the advising session. Advisor 
transcripts must be double-coded with student type. Add a memo if unclear about adherence to 
intervention or intervention design. 


References to whether and how the advisor used the advising toolbox during the advising 
session. Advisor transcripts must be double-coded with student type. Add a memo if unclear 
about adherence to intervention or intervention design. 


Description of whether and how multisemester course planning is addressed during the advising 
session. Includes description of whether a student has utilized education planning technology 
ahead of the advising session. Also use to capture the absence of discussion of education 
planning. Advisor transcripts must be double-coded with student type. Add a memo if unclear 
about adherence to intervention or intervention design. 


Description of whether and how career goals and/or career planning is addressed during the 
advising session. Also use to capture the absence of discussion of career goals and/or planning. 
Advisor transcripts must be double-coded with student type. Add a memo if unclear about 
adherence to intervention or intervention design. 


Description of whether and how referrals were made to other services during the advising 
session. Advisor transcripts must be double-coded with student type. Add a memo if unclear 
about adherence to intervention or intervention design. 
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